1. Field of the Invention
This invention relates to restoration based super-resolution for combining a video sequence of low-resolution noisy blurred images to produce a higher resolution image or video sequence, and more particularly to a technique for resolution enhancement without regularization up to a maximum enhancement factor r for a given video sequence that supports both integer and real-valued r.
2. Description of the Related Art
In defense and other applications camera systems produce a sequence of low-resolution (LR) noisy blurred images. The optical/mechanical/electrical imaging characteristics of the camera itself, in addition to atmospheric and motion blurring, limit the resolution and noise performance of the captured images. Super-resolution combines the low-resolution (LR) noisy blurred images and produces either a higher resolution (HR) image or sequence of HR images. Super-resolution techniques are based on an electro-optical model of the imaging system and rely on temporal motion of the camera from frame-to-frame so that the LR images contain similar but not identical information. Super resolution can significantly increase image resolution without changing optical/mechanical/electrical imaging characteristics of the camera.
A dynamic scene with continuous intensity distribution X(x, y) is seen to be warped at the camera lens because of the relative motion between the scene and camera. The images are blurred both by atmospheric turbulence and the camera lens by continuous point spread functions Hatm(x,y) and Hcam(x,y), respectively. The images are discretized at the digital video camera resulting in a digitized noisy frame Y(m,n). This forward model is represented by following:Y(m,n)=[Hcam(x,y)**F(Hatm(x,y)**X(x,y))]!+V(m,n)  (1)where ** is the two-dimensional convolution operator, F is the warping operator, ! is the discretizing operator, V(m,n) is the system noise, and Y(m,n) is the resulting discrete noisy and blurred low-resolution image. Based upon the model, high resolution (HR) image, X(x,y), can be estimated from a sequence of low resolution (LR) noisy images, Yk(m,n).
The general matrix super-resolution formulation from (1) can be simplified asYk=DkHk,camFHk,atmX+Vk k=1, . . . , L  (2)where the [r2M2×r2M2] matrix Fk is the geometric motion operator between the HR frame X (of size [r2M2×1] and the kth LR frame Yk (of size [M2×1]) and r is the resolution enhancement factor. The camera is modeled by the [r2M2×r2M2] blur matrix Hk,cam the atmosphere blur is represented by [r2M2×r2M2] matrix Hk,atm, and [M2×r2M2] matrix Dk represents the decimation operator. The [M2×1] vector Vk is system noise and L is the number of available LR frames. For a specific camera, Hk,cam will be a constant matrix. In many cases, atmosphere noise is relatively smaller than camera blur so Hk,atm can be ignored.
Using a Maximum Likelihood approach, a minimization criterion and iterative form of the estimation is presented as follows:
                    X        =                                            Arg              ⁢                                                          ⁢              Min                                      X              ^                                [                                    ∑                              k                =                1                            L                        ⁢                                                                                                                      D                      k                                        ⁢                                          H                      k                                        ⁢                                          F                      k                                        ⁢                                          X                      n                                                        -                                      Y                    k                                                                              2                                ]                                    (        3        )            Where Hk=Hk,cam. Since it depends on sensor only, H=Hk
                              X                      n            +            1                          =                              X            n                    -                      λ            (                                          ∑                                  k                  =                  1                                L                            ⁢                                                D                  k                  T                                ⁢                                  H                  T                                ⁢                                                      F                    k                    T                                    ⁡                                      (                                                                                            D                          k                                                ⁢                                                  HF                          k                                                ⁢                                                  X                          n                                                                    -                                              Y                        k                                                              )                                                                        )                                              (        4        )            2here λ is an iteration factor. Since DTD=1, FkTFk=1, and assume HFkXn=FkHXn (either shifting first or blurring first won't change LR images much)
                              X                      n            +            1                          =                              X            n                    -                      λ            ⁢                                                  ⁢                                          H                T                            (                                                ∑                                      k                    =                    1                                    L                                ⁢                                  (                                                            HX                      n                                        -                                                                  D                        k                        T                                            ⁢                                              F                        k                        T                                            ⁢                                              Y                        k                                                                              )                                            )                                                          (        5        )            
The matrix MLE is initialized for X0 using, for example, a Weiner filter and than iterated until Xn+1−Xn satisfies a convergence criterion.
This formulation of the MLE for super resolution based image restoration has a few well known deficiencies regarding the selection of the enhancement factor “r” that both prevent its direct application to real imagery in fielded restoration systems and limit its resolution enhancement capability and flexibility.
If the selected “r” value is too big, a significant number of pixels in the restored high-resolution image will have no value assigned to them from one or more of the LR images. This can induce failure of the restoration process (‘ringing’ in the HR image). Therefore, known super resolution systems employ some form of ‘regularization’ (spatial smoothing) to ensure the system is robust (ringing does not occur). Sina Farsiu et al. “Fast and Robust Multiframe Super Resolution”, IEEE Transactions on Image Processing, Vol. 13, No. 10, October 2004 pp. 1327-1344 addresses the issue of regularization in section “C. Robust Regularization” on p. 1331 stating that super-resolution is an ill-posed problem. For the under-determined cases (i.e., when fewer the r2 frames are available) there exist an infinite number of solutions which satisfy the general super resolution matrix formulation. The solution for square and over-determined cases is not stable, which means small amounts of noise in measurements will result in large perturbations in the final solution. Farsiu states “Therefore, considering regularization in super-resolution algorithms as a means for picking a stable solution is very useful, if not necessary.” A regularization term compensates the missing measurement information with some general prior information about the desirable HR solution, and is usually implemented as a penalty factor in the generalized minimization cost function. The regularization is performed whether a particular r is too big or not for a particular sequence of LR images because that determination is unknown. Regardless of the sophistication of a particular regularization approach, regularization inherently removes some high frequency content which tends to blur sharp edges in the imaged scene and create other artifacts.
Another deficiency is that all existing restoration based super-resolution approaches use the matrix formulation of the MLE in equation 5, which is inherently limited to integer-valued enhancement factors. Downsampling matrix D is only valid for integer values of r. Thus these systems are often un-robust.